Analysing a binary vs. scale variable

Test: independent samples t-test

From the descriptive statistics in the impression and visualisation section, we noticed a difference in means between male and female students. However, this was just a sample, and would this mean that there is also a difference in the population. To find out, we’d need to perform a statistical test.

The most commonly used test is probably the Student t-test (Student, 1908). However, this test does not perform well if the variances in each group are different. The Welch test (Welch, 1938 , 1947) is designed to use if the variances are not equal, and does not differ much from the Student t-test if the variances are the same. Ruxton (2006) compared the Welch t-test and the Student’s t-test and concluded that: “the unequal variance t-test should always be used in preference to the Student's t-test” (p. 690).

In the example the independent samples Welch t-test results in a significance of .499. This means that if there would not be any difference in means in the population, the chance of a difference as we had in the sample, or an even bigger difference, is .499. Usually this is considered a ‘high’ chance (usually anything above .05 is considered ‘high’). Since the chance of a result as in our sample (or even larger) is high, the assumption that the means in the population are equal could be true. We therefor can conclude that there is no significant difference (or slightly better: not enough evidence to claim a difference).

We could report the results as shown below:

A independent-samples Welch t-test indicated that there is insufficient evidence to claim a significant difference in the mean grades given to the course by the Male students (M = 60, SD = 19) and the Female students (M = 54, SD = 26), t(14.16) = .694, p = .499.

Click here to see how to perform an independent samples Welch t-test with SPSS, R studio, Excel, Python, or Manually

With SPSS

With R studio

With Excel

The add-on Data Analysis can be used with Excel to perform this test, but it can also be done without.

without Data Analysis

with Data Analysis

With Python

Manually

The video below explains how to perform an independent samples Welch t-test manually. Below the video the formula's are also written out, and an example, in case you prefer to read.

The formula's

The formula for the t-value with a Welch t-test is given by:

$t=\frac{\bar{x}_{1}-\bar{x}_{2}}{SE}$

In this formula $\bar{x}_{i}$ is the mean (average) of categori i. In formula notation:

$\bar{x}_{i}=\frac{\sum_{j=1}^{n_{i}}x_{i,j}}{n_{i}}$

n_i the number of cases in category i, and x_i,j the j-th score of category i.

SE is the standard error. The standard error has the following formula:

$SE=\sqrt{\frac{s_{1}^{2}}{n_{1}}+\frac{s_{2}^{2}}{n_{2}}}$

Where $s_{i}^{2}$ is the sample variance of category i. In formula notation:

$s_{i}^{2}=\frac{\sum_{j=1}^{n_{i}}\left (x_{i,j}-\bar{x}_{i} \right )^{2}}{n_{i}-1}$

The degrees of freedom is given by:

An example.
Note: a different example than the one used in the rest of this section, to keep calculations a bit shorter.

Given are the scores of males (category 1) and females (category 2):

$X_{1}=\left$8,3,2,1,1\right$$

$X_{2}=\left$7,7,5,3,9,8\right$$

By simply counting the number of cases, we can determine that there were 5 male scores, and 6 female scores. So we have:

$n_{1}=5, n_{2}=6$

Now, let's fill out the formula for the mean, for the males and females:

$\bar{x}_{1}=\frac{\sum_{j=1}^{n_{1}}x_{1,j}}{n_{1}}=\frac{\sum_{j=1}^{5}x_{1,j}}{5}=\frac{8+3+2+1+1}{5}=\frac{15}{5}=3$

$\bar{x}_{2}=\frac{\sum_{j=1}^{n_{2}}x_{2,j}}{n_{2}}=\frac{\sum_{j=1}^{6}x_{1,j}}{6}=\frac{7+7+5+3+9+8}{6}=\frac{39}{6}=\frac{13}{2}=6.5$

With the means for each category, we can now determine the variance for each category:

$s_{1}^{2}=\frac{\sum_{j=1}^{n_{1}}\left(x_{1,j}-\bar{x}_{1}\right)^{2}}{n_{1}-1}=\frac{\sum_{j=1}^{5}\left(x_{1,j}-3\right)^{2}}{5-1}$

$=\frac{\left(8-3\right)^{2}+\left(3-3\right)^{2}+\left(2-3\right)^{2}+\left(1-3\right)^{2}+\left(1-3\right)^{2}}{5-1}=\frac{\left(5\right)^{2}+\left(0\right)^{2}+\left(-1\right)^{2}+\left(-2\right)^{2}+\left(-2\right)^{2}}{5-1}$

$=\frac{25+1+4+4}{5-1}=\frac{34}{4}=\frac{17}{2}=8.5$

And for the females

$s_{2}^{2}=\frac{\sum_{j=1}^{n_{2}}\left(x_{2,j}-\bar{x}_{2}\right)^{2}}{n_{2}-1}=\frac{\sum_{j=1}^{6}\left(x_{2,j}-\frac{13}{2}\right)^{2}}{6-1}$

$=\frac{\sum_{j=1}^{6}\left(7-\frac{13}{2}\right)^{2}+\left(7-\frac{13}{2}\right)^{2}+\left(5-\frac{13}{2}\right)^{2}+\left(3-\frac{13}{2}\right)^{2}+\left(9-\frac{13}{2}\right)^{2}+\left(8-\frac{13}{2}\right)^{2}}{6-1}$

$=\frac{\sum_{j=1}^{6}\left(\frac{14}{2}-\frac{13}{2}\right)^{2}+\left(\frac{14}{2}-\frac{13}{2}\right)^{2}+\left(\frac{10}{2}-\frac{13}{2}\right)^{2}+\left(\frac{6}{2}-\frac{13}{2}\right)^{2}+\left(\frac{18}{2}-\frac{13}{2}\right)^{2}+\left(\frac{16}{2}-\frac{13}{2}\right)^{2}}{6-1}$

$=\frac{\sum_{j=1}^{6}\left(\frac{1}{2}\right)^{2}+\left(\frac{1}{2}\right)^{2}+\left(-\frac{3}{2}\right)^{2}+\left(-\frac{7}{2}\right)^{2}+\left(\frac{5}{2}\right)^{2}+\left(\frac{3}{2}\right)^{2}}{6-1}$

$=\frac{\frac{1}{4}+\frac{1}{4}+\frac{9}{4}+\frac{49}{4}+\frac{25}{4}+\frac{9}{4}}{6-1}=\frac{\frac{1+1+9+49+25+9}{4}}{6-1}$

$=\frac{\frac{94}{4}}{6-1}=\frac{\frac{47}{2}}{5}=\frac{47}{10}=4.7$

Now we have all the information to fill out the other formulas. First the Standard Error:

$SE=\sqrt{\frac{s_{1}^{2}}{n_{1}}+\frac{s_{2}^{2}}{n_{2}}}=\sqrt{\frac{\frac{17}{2}}{5}+\frac{\frac{47}{10}}{6}}=\sqrt{\frac{17}{10}+\frac{47}{60}}=\sqrt{\frac{17\times6}{10\times6}+\frac{47}{60}}$

$=\sqrt{\frac{102}{60}+\frac{47}{60}}=\sqrt{\frac{102+47}{60}}=\sqrt{\frac{149}{60}}=\frac{1}{60}\sqrt{149\times60}=\frac{1}{60}\sqrt{8940}=\frac{1}{60}\sqrt{4\times2235}=\frac{2}{60}\sqrt{2235}=\frac{1}{30}\sqrt{2235}\approx1.576$

The t-value is therefor:

$t=\frac{\bar{x}_{1}-\bar{x}_{2}}{SE}=\frac{3-\frac{13}{2}}{\frac{1}{30}\sqrt{2235}}=\frac{\frac{6}{2}-\frac{13}{2}}{\frac{1}{30}\sqrt{2235}}=\frac{\frac{6-13}{2}}{\frac{\sqrt{2235}}{30}}$

$=\frac{\frac{-7}{2}}{\frac{\sqrt{2235}}{30}}=\frac{-7\times30}{2\sqrt{2235}}=\frac{-210}{2\sqrt{2235}}=\frac{-105}{\sqrt{2235}}\approx-2.221$

And we can also determine the degrees of freedom:

$df=\frac{\left(\frac{s_{1}^{2}}{n_{1}}+\frac{s_{2}^{2}}{n_{2}}\right)^{2}}{\frac{\left(s_{1}^{2}\right)^{2}}{n_{1}^{2}\left(n_{1}-1\right)}+\frac{\left(s_{2}^{2}\right)^{2}}{n_{2}^{2}\left(n_{2}-1\right)}}=\frac{\left(\frac{\frac{17}{2}}{5}+\frac{\frac{47}{10}}{6}\right)^{2}}{\frac{\left(\frac{17}{2}\right)^{2}}{5^{2}\left(5-1\right)}+\frac{\left(\frac{47}{10}\right)^{2}}{6^{2}\left(6-1\right)}}$

$=\frac{\left(\frac{17}{10}+\frac{47}{60}\right)^{2}}{\frac{\frac{17^2}{2^2}}{25\left(4\right)}+\frac{\frac{47^2}{10^2}}{36\left(5\right)}} =\frac{\left(\frac{149}{60}\right)^{2}}{\frac{\frac{289}{4}}{100}+\frac{\frac{2209}{100}}{180}} =\frac{\frac{22201}{3600}}{\frac{289}{400}+\frac{2209}{18000}}$

$=\frac{\frac{22201}{3600}}{\frac{289\times45}{400\times45}+\frac{2209}{18000}} =\frac{\frac{22201}{3600}}{\frac{13005}{18000}+\frac{2209}{18000}} =\frac{\frac{22201}{3600}}{\frac{13005+2209}{18000}} =\frac{\frac{22201}{3600}}{\frac{15214}{18000}} =\frac{\frac{22201}{3600}}{\frac{7607}{9000}}$

$=\frac{\frac{22201}{3600}}{\frac{7607}{9000}} =\frac{22201\times9000}{3600\times7607} =\frac{22201\times5}{2\times7607} =\frac{111005}{15214}\approx7.296$

With the t-value and the degrees of freedom, the significance can be determined by using a t-distribution table.

When there is a significant difference, we might also want to check the ‘size’ of the difference. For example if we would have had a difference of 0.0003 in grades, then with extreme large sample sizes this could still be significant, but not really relevant. To measure the size of the difference we would need a so-called effect size, which is the topic for the next section.

Binary vs Scale

Reporting

Google adds